A general video compression codec uses a method of compressing a video by applying a transform to a residual signal obtained via intra prediction or inter prediction and then performing quantization and entropy coding on a transform coefficient. A transform shape is a square shape, and a a discrete cosine transform (DCT) kernel or a kernel in which a DCT is approximated to an integer transform is generally used as a transform kernel. For example, recently, a high efficiency video coding (HEVC) codec adopts a square integer transform having a size of 4×4, 8×8, 16×16, or 32×32, and the square integer transform is obtained by approximating a DCT. However, HEVC uses an integer transform obtained by approximating a discrete sine transform (DST) only for a residual signal having a size of 4×4 exclusively obtained via intra prediction.
Recently, the joint video experts team (JVET) and others have actively explored technology for next generation video codec standardization since HEVC, and new transform technologies have been introduced. For example, adaptive multiple transformation (AMT) is a method of selecting a kernel to be used for a current transform unit (TU) from among a plurality of pre-defined candidate transform kernels, applying the selected kernel to the current TU, and additionally transmitting information about the selected kernel. In this case, the set of candidate transform kernels are fixed in a coding unit (CU) to which inter prediction is applied, and vary according to an inter prediction mode in a CU to which intra prediction is applied. In addition, a number of secondary transform technologies of applying a transform again to a transform coefficient obtained after primary transformation have been proposed. For example, rotational transformation (ROT) involves splitting a transform coefficient into 4×4 units, selecting one from among pre-determined secondary transforms, and applying the selected secondary transform to the 4×4 units. In non-separable secondary transformation (NSST) that operates like ROT, a transform kernel is non-separable and secondary transform kernel candidates to be applied vary based on an intra prediction mode like in AMT.